Eye searching method and eye state detection apparatus and eye searching apparatus using the same

ABSTRACT

The present invention relates to an eye searching method and an eye state detection apparatus and an eye searching apparatus using the same. The method comprises the steps of: defining each center point of two nostrils from a face image; computing a distance D between the two center points of the two nostrils and a midpoint between the two center points of the two nostrils; determining a reference point that has a horizontal distance k1×D and a vertical distance k2×D spaced apart from the midpoint, wherein k1=1.6˜1.8, k2=1.6˜1.8; and defining a rectangle around the reference point as its center, wherein the rectangle has a width and a length greater than the width, and the rectangle surrounds the eye of the face image. Accordingly, the present invention can quickly and accurately find out a certain part of the eye from the face image.

BACKGROUND OF INVENTION

1. Field of Invention

The invention relates to an image processing technology, and moreespecially to a technology that can quickly and accurately find out aneye from a face image.

2. Related Prior Art

Technology related to eye state identification, such as detecting adriver's eye state to identify whether he/she is dozing off whiledriving, or detecting eye movement of a person suffering fromAmyotrophic lateral sclerosis (ALS) for control purpose, has been widelyknown. Also, there are lots of patent applications, for example, TaiwanPatent No. 436436, I349214, M416161, TW 201140511, U.S. Pat. No.8,094,883, U.S. Pat. No. 8,184,008, CN101196993 and so on, whichdescribe an image processing method for eye state identification. Theaforementioned technology comprises several steps, including capturing auser's face by a camera device to obtain images of the user's face,processing these images by an eye-identification device, and thendetermining the eye state of the user based on the processed results.The step of processing each of the images by the eye-identificationdevice mentioned above further comprises the following sub-steps insequence: finding out a face area, obtaining an eye area according tothe face area that is found, attaining an specific area of the eye basedon the eye area (such as upper eyelid, lower eyelid, or pupil), anddetermining the eye state of the user (i.e. whether the eye is opened orclosed) according to variation of the specific area. However, how tofind out the eye area on the face image quickly and precisely is one ofthe significant issues.

A conventional eye detection apparatus disclosed in China Patent No.101196993 includes means for finding out positions of nostrils from aface image, means for establishing an eye searching area based on thepositions of the nostrils, and means for attaining upper and lowereyelids from the eye searching area. It is understood that the nostrilson the face image is black, which therefore can be easily and preciselyidentified. Accordingly, in comparison with the traditional way to findout the eye directly from the face image, the eye detection apparatuscan find out the eye more quickly and precisely.

Nevertheless, the eye searching area established by the conventional eyedetection apparatus is a rectangle area with a horizontal side and avertical side greater than the horizontal side, which covers a largearea including eyebrows and cheeks. This obviously results anotherproblems that the conventional eye detection apparatus would take muchmore time on finding out the upper/lower eyelids on the large eyesearching area.

SUMMARY OF INVENTION

The present invention is directed to a method and an apparatus, whichcan solve the aforementioned programs that the conventional eyedetection apparatus would take much more time on finding out a certainpart of the eye, such as the upper/lower eyelids on the large eyesearching area.

Specifically, the present invention relates to an eye state detectionapparatus, which comprises a storage unit, a camera unit and aprocessing unit. The camera unit is configured for capturing an originalimage of a user's face. The storage unit is configured for storing aprogram and the original image of the user's face. The processing unitis electrically connected with the camera unit and the storage unit, andis configured to load and execute the program to perform eye detection.Specifically, when the processing unit loads and executes the program,the program causes the processing unit to receive an original image,obtain a face image from the original image; define each center point oftwo nostrils respectively on the face image, compute a distance Dbetween the two center points of the two nostrils and determiningcoordinates of an initial point A (x1, y1), determine coordinates of areference point B (x2, y2) according to the distance D and the initialpoint A (x1, y1), define a rectangle on the face image according to thecoordinates of the reference point B (x2, y2), and obtain an eye imageof the user's eye from an area of the rectangle. The above-mentioned theinitial point A (x1, y1) represents a midpoint between the two centerpoints of the two nostrils, and x2=x1+k1×D, y2=y1+k2×D, k1=1.6˜1.8,k2=1.6˜1.8, preferably k1=k2. The reference point B (x2, y2) representsa middle point of the rectangle. The rectangle has a width in a verticaldirection and a length in a horizontal direction that is greater thanthe width; and the eye image include the user's eye thereon.

Preferably, the program further cause the processing unit to beprogrammed to perform an eye state estimation based on the eye image toobtain an upper eyelid feature on the eye image, determine a curvedlevel of the upper eyelid feature and generate an eye state data basedon the determined result, wherein the eye state data represents eitheropening eye state or closing eye state.

The present invention is further directed to an eye searching apparatus,comprising a processing unit and a program, wherein the processing unitis configured to execute the program to perform eye detection.Specifically, when the processing unit executes the program, the programcauses the processing unit to be programmed to run the following stepsof the eye searching method and state estimation step. The eye stateestimation step is to obtain the upper eyelid feature on the eye image,determine a curvature level of the upper eyelid feature, and generate aneye state data representing either opening eye state or closing eyestate. The steps includes: a) defining each center point of two nostrilsrespectively on a face image; b) computing a distance D between the twocenter points of the two nostrils and a midpoint between the two centerpoints of the two nostrils; and c) obtaining an eye from the face imagebased on the distance D and the midpoint. Preferably, the aforementionedstep c) further includes: determining a reference point havingcoordinates (k1×D, k2×D), and defining a rectangle around the referencepoint as its center, wherein k1=1.6˜1.8, k2=1.6˜1.8, the rectangle has awidth in a vertical direction and a length in a horizontal directiongreater than the width, and the rectangle surrounds the eye of the faceimage. The k1 mentioned above is preferably equal to k2.

Compared with the prior art, it is believed that the eye image or theabove-mentioned rectangle obtained by the present invention includes theeye thereon, which has smaller area than the eye searching area in theprior art. As such, it is easier for the processing unit of the presentinvention to quickly obtain a certain part of the eye by searching thesmaller area on the image.

Other features, objects, aspects and advantages will be identified anddescribed in detail below.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of an eye state detection apparatus inaccordance with an embodiment of the present invention;

FIG. 2 is a process flow diagram that illustrates instructions executedby the processing unit object in accordance with an embodiment of thepresent invention;

FIG. 3 is a perspective view of an image captured by the camera unit inaccordance with an embodiment of the present invention.

FIG. 4 is a perspective view of an eye image obtained form the image inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

With reference to FIG. 1, a block diagram of an eye state detectionapparatus is shown in accordance with an embodiment of the presentinvention. The eye state detection apparatus comprises a storage unit 1,a camera unit 2, a processing unit 3, an output unit 4 electricallyconnected to the processing unit 3, and an input unit 5 (that consistsof several buttons). The storage unit 1 consists of one or severalnon-volatile access memory, which stores a program and the originalimage of the user's face captured by the camera unit 1. The program hasa set of eye detection program code 10, preferably further includes aset of eye state estimation program code 11. The camera unit 1 isconfigured to capture a user's face to produce several successiveoriginal images and send the images to the storage unit 1 for temporallystoring thereon.

The camera unit 2 is preferably a rotatable lens (not shown in thedrawings) that can be rotated at any angle and in any direction,allowing the lens to face the user's, face at a certain elevation angle,such as an elevation angle of 45 degree. As such, the nostrils on eachof the face images captured by the camera unit 2 can be clearly shown.This means that the legibility of the nostrils on each of the faceimages can be significantly increased, which benefits the follow-upnostril searching process. In general, the camera unit 2 furtherincludes an illumination device for emission of light to ensure greatquality of the face image.

The processing unit 3 is electrically connected to the camera unit 2 andthe storage unit 1, which includes at least a central processing unit(CPU, not shown) and a random-access memory (RAM, not shown). When theprocessing unit 3 loads and executes the eye searching program code 10,the processing unit 3 is programmed to run the following tasks, as shownin FIGS. 2 and 3, including:

a) receiving an original image 6 (block a of FIG. 2), wherein theoriginal image 6 is obtained by capturing the user's face through thecamera unit 2; and the original image 6 includes not only the user'sface, but also some unnecessary parts that are needed to be removed,such as hair, neck and background;

b) obtaining a face image 61 from the original image 6 (block b of FIG.2), which can be performed by Adaboost algorithm and other imageprocessing techniques, Wherein the unnecessary portions mentioned abovehave partially or completely been removed from the illustrative faceimage 61 preferably;

c) defining each center point 62 of two nostrils respectively on theface image 61 (block c of FIG. 2), wherein the way to define the twocenter points of the two nostrils on the face image has been disclosedin the prior art, which would not be illustrated hereinafter; the centerpoints 62 of the two nostrils can be respectively defined by creating anintersection of the longest transverse and longest longitudinal axes ofeach nostril area as the nostril area on the face image 61 is obviouslydarker than the other areas;

d) computing a distance D between the two center points 62 of the twonostrils and determining coordinates of an initial point A (x1, y1)(block d of FIG. 2), wherein the coordinates of the initial point A (x1,y1) represents a midpoint between the two center points of the twonostrils;

e) determining coordinates of a reference point B (x2, y2) according tothe distance D and the initial point A (x1, y1) (block e of FIG. 2),wherein x2=x1+k1×D, y2=y1+k2×D, k1=1.6˜1.8, k2=1.6˜1.8, and preferably,k1=k2; it is noted that the reference point B (x2, y2) locates at ornear to a midst point of an eye of the face image 61 according to thereal experiential results; wherein similarly, coordinates of anotherreference point C (x3, y3) can be further determined according to thedistance D and the initial point A (x1, y1) if necessary, whereinx3=x1−k1×D, y3=y1−k2×D; and it is noted that the illustrative referencepoint C (x3, y3) locates at or near to a midst point of another eye ofthe face image 61;

f) defining a rectangle R1 on the face image 61 according to thecoordinates of the reference point B (x2, y2) (block f of FIG. 2);wherein the coordinates of the reference point B (x2, y2) represents amiddle point of the rectangle R1; the rectangle R1 has a length W1 in ahorizontal direction and a width W2 in a vertical direction, W1=30˜50pixel, W2=15˜29 pixel, W1>W2, preferably W1=40 pixel, W2=25 pixel; andwherein similarly, another rectangle R2 that is the same as therectangle R1 can be further defined according to the coordinates of thereference point C (x3, y3) if necessary, wherein the coordinates of thereference point C (x3, y3) represents a middle point of the rectangleR2; and

g) obtaining an eye image 63 including the user's eye thereon from anarea of the rectangle R1 (block g of FIG. 2), as shown in FIG. 4;wherein it is noted that another eye image from an area of the rectangleR2 can be further obtained if necessary.

It is understood that the rectangle R1 from bock f surrounds one eye ofthe face image 61 according to the real results. Moreover, it isbelieved that either the eyebrows above the eye or the cheeks below theeye does not locate within the illustrative rectangle R1 (alternativelymay a few part of the eyebrows locate within the rectangle R1). Also,similarity, the other rectangle R2 surrounds the other eye of the faceimage 61, which may exclude the eyebrows and the cheeks thereon. Thismeans that the eye image 61 from block g includes the eye thereon, andan area of any one of the eye images of the present invention is muchsmaller than the eye searching area in prior art.

The processing unit 3 would further continue loading and executing theeye state estimation program code 11 after obtaining the eye image 63 orthe two eye images 63. The eye state estimation program code 11 causesthe processing unit 3 to be programmed to run an eye state estimationstep to define a certain part of the eye on the eye image 63, such asupper/lower eyelid feature or pupil feature, and generate an eye statedata according to detected result of the certain part of the eye. Theillustrative eye state data includes either “0” representing opening eyestate or “1” representing closing eye state, for example. In the presentinvention, the eye state data is obtained by determining a curvaturelevel of the upper eyelid. Specifically, when the eye is opening, theupper eyelid shows a cured line relative to the horizontal direction; onthe contrary, when the eye is closing, the upper eyelid shows anapproximate straight line relative to the horizontal direction. Afterprocessing the eye image 63 by an image processing technology, an uppereyelid image can be created from the upper eyelid of the eye image 63.If the eye on the eye image 63 is opening, the upper eyelid image showsa parabolic curved line, which indicates that a focal length of theparabolic curved line can be calculated by the processing unit 3according to the parabolic equation. However, if the eye on the eyeimage 63 is closing, the upper eyelid image shows a straight line, whichobviously means that the focal length thereof extends into infinity. Ina word, the processing unit 3 processes the eye image 63 to generate afocal length, the different focal lengths represent different curvaturelevel of the upper eyelid, and different curvature levels of the uppereyelid indicate different eye state. More specifically, when the focallengths obtained from the processing unit 3 by processing successiveimages gradually increase from a certain value to infinity, the eye onthe captured image is then determined as the closing eye state andtherefore the eye state data “1” representing closing eye state isgenerated. On the contrary, when the focal lengths obtained byprocessing successive images gradually decrease from infinity to acertain value, the eye on the captured image is then determined as theopening eye state and therefore the eye state data “0” representingopening eye state is generated.

Information needed for carrying out the aforementioned tasks by theprocessing unit 3 or generated from the processing unit 3, such as theface image or the eye image, is stored in the storage unit 1. Theinformation may be temporally stored or permanently saved in the storageunit 1 depending on the needs.

As illustrated above, it is understood that the eye state detectionapparatus can take an image of the user's face by the camera unit 2,process each of the images by the processing unit 3, and generate an eyestate data depending upon the results from processing unit 3 byprocessing each of the images. For example, if the eye state data shows“1” representing closing eye state, or the eye state data “1” shows veryoften with increasing frequency higher than a threshold, the processingunit 3 determines that the user is dozing off. At this time, the outputunit 4 is driven to transmit an alarm message, such as an alarm soundoutput from a loudspeaker of the output unit 4. Preferably, the outputunit 4 further includes a display device (such as a touched panel) fordisplaying the alarm message or other related information thereon (suchas an the human-computer interface for setting operations).

According to the above descriptions from task a to task b, theprocessing unit 4 actually is to perform an eye searching method,including steps of:

defining each center point of two nostrils on a face image respectively;

computing a distance D between the two center points of the two nostrilsand a midpoint between the two center points of the two nostrils;

determining a reference point having the coordinates (k1×D, k2×D),wherein k1=1.6˜1.8, k2=1.6˜4.8, preferably k1=k2; and

defining a rectangle around the reference point as its center, whereinthe rectangle has a length W1 in a horizontal direction and a width W2in a vertical direction, W1=30˜50 pixel, W2=15˜29 pixel and W1>W2,preferably W1=40 pixel and W2=25 pixel.

Compared with the prior art, it is believed that the eye image or theabove-mentioned rectangle obtained by the present invention includes theeye thereon, which has smaller area than the eye searching area in theprior art. As such, it is easier for the processing unit 3 of thepresent invention to quickly find out a certain part of the eye bysearching the smaller area on the image.

It will be appreciated that although a particular embodiment of theinvention has been shown and described, modifications may be made. It isintended in the claims to cover such modifications which come within thespirit and scope of the invention.

The invention claimed is:
 1. An eye state detection apparatus,comprising: a camera unit configured for capturing an original image ofa user's face; a storage unit configured for storing a program and theoriginal image of the user's face; and a processing unit electricallyconnected with the camera unit and the storage unit, the processing unitconfigured to load and execute the program to receive the originalimage, obtain a face image from the original image, define each centerpoint of two nostrils on the face image, compute a distance D betweenthe two center points of the two nostrils and determine coordinates ofan initial point A (x1, y1) representing a midpoint between the twocenter points of the two nostrils, determine coordinates of a referencepoint B (x2, y2) according to the distance D and the initial point A(x1, y1), define a rectangle on the face image according to thecoordinates of the reference point B (x2, y2), and obtain an eye imageof the user's eye from an area of the rectangle; wherein x2=x1+k1×D,y2=y1+k2×D, k1=1.6˜1.8, k2=1.6˜1.8, and the coordinates of the referencepoint B (x2, y2) represents a middle point of the rectangle.
 2. Theapparatus of claim 1, wherein k1=k2.
 3. The apparatus of claim 1,wherein the processing unit is further to perform an eye stateestimation based on the eye image to obtain an upper eyelid feature onthe eye image, determine a curved level of the upper eyelid feature andgenerate an eye state data based on the determined result, wherein theeye state data represents either opening eye state or closing eye state.4. The apparatus of claim 3, wherein the eye state estimation isperformed by the process unit, in which the upper eyelid feature on theeye image is presented as a parabolic curved line with a focal length,the curved level of the upper eyelid feature is determined by the focallength, and if focal lengths obtained by processing successive eyeimages gradually increase from a certain value, the eye is thendetermined as in the closing eye state, and if the focal lengthsgradually decrease to a certain value, the eye is then determined as inthe opening eye state.
 5. The apparatus of claim 4, wherein if the eyestate data shows that the eye is in the closing eye state with anoccurring frequency higher than a threshold, the processing unitdetermines that the user is dozing off.
 6. An eye searching method,comprising the following steps of: a) defining each center point of twonostrils on a face image respectively; b) computing a distance D betweenthe two center points of the two nostrils and a midpoint between the twocenter points of the two nostrils; and c) obtaining an eye on the faceimage based on the distance D and the midpoint; wherein step c furthercomprises: determining a reference point having the coordinates (k1×D,k2×D), wherein k1=1.6˜1.8, k2=1.6˜1.8; and defining a rectangle aroundthe reference point as its center, wherein the rectangle has a width ina vertical direction and a length in a horizontal direction greater thanthe width, and the rectangle surrounds the eye of the face image.
 7. Themethod of claim 6, wherein k1=k2.
 8. An eye searching apparatuscomprising a processing unit and a program, wherein the processing unitis configured to execute the program to define each center point of twonostrils respectively from a face image, compute a distance D betweenthe two center points of the two nostrils and a midpoint between the twocenter points of the two nostrils, and obtain an eye from the face imagebased on the distance D and the midpoint; and wherein the processingunit is further configured to determine a reference point has thecoordinates (k1×D, k2×D), and define a rectangle around the referencepoint as its center, wherein k1=1.6˜1.8, k2=1.6˜1.8, the rectangle has awidth in a vertical direction and a length in a horizontal directiongreater than the width, and the rectangle surrounds the eye of the faceimage.
 9. The apparatus of claim 8, wherein k1=k2.